Yes, the oxford english dictionary defines racism in terms of belief: the prejudice and discrimination must be based on belief. You must have the belief that members of a race etc etc. If you discriminate against someone without the belief that your race is superior, that is not racism. If you believe that all members of a race possess characteristics etc. that don't distinguish it as superior or inferior, that is not racism.
Question: How exactly do I exactly access someones beliefs to determine whether or not they embrace racist ideals?
So sentences like "go back to where you came from" are ambiguous. We need to use context.
However, context shouldn't matter either! Because if we don't have direct access to others beliefs, and we use OED as our guide, we really can't say anything about anyone being racist.